Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax. Experiments demonstrate improved probabilistic predictions according to multiple measures (confidence-ECE, classwise-ECE, log-loss, Brier score) across a wide range of datasets and classifiers. Parameters of the learned Dirichlet calibration map provide insights to the biases in the uncalibrated model.

POST REBUTTAL COMMENTS This did not affect my decision and score, but the citations on calibration should be improved. The oldest work on calibration that the authors cite is from 2000 (Platt). The definitions of calibration/calibration error, and many of the key ideas were proposed several decades ago by statisticians and meteorologists like Brier, Murphy, Winkler, Deegroot, Fienberg. The calibration error metric proposed there is different from ECE (root-mean-squared instead of mean absolute value), it would be good if the authors can mention that the RMS calibration error is another possible metric. I've included some pointers to the literature below (there are many other papers, but it should be OK to just cite a few): - Verification of forecasts expressed in terms of probability.

The paper completes the picture of post-training calibration by proposing Dirichlet calibration as a natural generalization of Beta calibration to the multi-class setting, and showing the connection between it and matrix scaling in the context of neural net models. The comprehensive experiments with both deep neural nets and non-neural models comparing a variety of post-training calibration techniques are also a strong point of the paper and was appreciated by all reviewers. On the negative side, the results are mixed with performance differences between the new techniques and other approaches being rather small. The authors should incorporate the reviewers' comments (R4 gave very detailed and thoughtful post-rebuttal comments). In particular the authors should: - cite older calibration work from statistics (see R4 comments for references).

Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax.

Class probabilities predicted by most multiclass classifiers are uncalibrated, often tending towards over-confidence. With neural networks, calibration can be improved by temperature scaling, a method to learn a single corrective multiplicative factor for inputs to the last softmax layer. On non-neural models the existing methods apply binary calibration in a pairwise or one-vs-rest fashion. We propose a natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification. It is easily implemented with neural nets since it is equivalent to log-transforming the uncalibrated probabilities, followed by one linear layer and softmax.